An instrumental least squares support vector machine for nonlinear system identification

Least-Squares Support Vector Machines (LS-SVM’s), originating from Stochastic Learning theory, represent a promising approach to identify nonlinear systems via nonparametric estimation of nonlinearities in a computationally and stochastically attractive way. However, application of LS-SVM’s in the identification context is formulated as a linear regression aiming at the minimization of the l2 loss in terms of the prediction error. This formulation corresponds to a prejudice of an auto-regressive noise structure, which, especially in the nonlinear context, is often found to be too restrictive in practical applications. In [1], a novel Instrumental Variable (IV) based estimation is integrated into the LS-SVM approach providing, under minor conditions, a consistent identification of nonlinear systems in case of a noise modeling error. It is shown how the cost function of the LS-SVM is modified to achieve an IV-based solution. In this technical report, a detailed derivation of the results presented in Section 5.2 of [1] is given as a supplement material for interested readers. 1 IV in the dual form Consider the primal minimization problem (eq. (52) in [1]): min θ∈Rnθ 1 2 θ⊤θ + γ 2N2 ∥∥Γ⊤E∥∥2 l2 , (1a) s.t. e(k) = y(k)− φ⊤(k)θ, k = 1, . . . , N, (1b) φi (0)θi = 0, i = 1, . . . , ng. (1c)